Dynamic variation of output media signal in response to input media signal

ABSTRACT

A first input signal, which has a first playback sequence, is resequenced based upon a characteristic of a second input signal. The resequencing occurs analyzing the characteristic in the second input signal, and modifying the first playback sequence of the first input signal based upon the analysis of the characteristic to generate a second playback sequence. Finally, a third signal is output using the second playback sequence.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. provisional patentapplication No. 60/322,944 entitled “System and Method for CreatingMultimedia”, filed Sep. 15, 2001, in the names of NEUMAN et al., thedisclosure of which is expressly incorporated herein by reference in itsentirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to multimedia. More specifically, thepresent invention relates to creating an output media signal based uponan input media signal.

2. Background Information

The combination of audio and video has become a popular means ofentertainment in film, television, video games, computers, PDAs, theinternet, kiosks, karaoke, ambient environments, and live events. It isextremely desirable to correlate a moving image to a sound track, orvice versa. This technique is demonstrated in film, television, musicvideos, video games, web sites, cartoons, clubs, events, concerts, lightshows, CD-ROMs, art installations, etc.

Current technology requires the user to manually edit a moving picturewith a soundtrack, either by editing frames or altering the playbackrate of the audio or video. This process requires highly skilled usersto work with complex editing tools and understand music structure so asto edit in synchronicity with a sound track. This traditional editingprocess is both very expensive and time consuming. In fact, one of themore costly and time consuming aspects of professional production offilm, video, and multimedia is synchronizing audio and video sources.

Recent developments in home consumer level systems capable of playingand manipulating video has created a demand for an editing system thatcreates motion pictures synchronized to an audio track. This process istoo complicated for the average user to perform utilizing currenttechnologies. Under the status quo, a user has to correlate an audiotrack to moving images by either:

1. Simultaneously capturing audio and motion pictures, which hassignificant problems; In order to have a correlation between the imagesand the audio, elaborate pre-production choreography is required;Another disadvantage of this simultaneous audio/video capture is thatthe audio and video become dependent; This restricts the user fromeasily changing either the audio or video of the synchronized piece; or

2. Manually editing a motion picture, cutting scenes in relation tochanges in a soundtrack; This technique requires the user to detectappropriate cues in a soundtrack, such as a snare drum, and then editthe picture at that moment in time; Changes in the picture or audioportions of the program require manually re-setting these in and outpoints, or utilizing a semi-automated technology whereby the user has tomanually correlate the sound cue to the desired frame and lock therelationship in the time line.

At live events, video is frequently used to enhance the viewer'sexperience. Current technology allows only basic automated visualeffects in response to audio. This is demonstrated by strobing lightsand lasers triggered by audio characteristics or, in the case of video,image manipulation of color, hues or similar attributes. More complexcorrelations between audio and motion pictures typically utilize aprocess in which a user pre-programs video effects, and manuallytriggers such effects through an input device, such as a keyboard,mouse, or trackball. The status quo lacks tools for generating real timevideo content derived from dynamic media signals, which do not requireanalysis or input from a user.

The creation of a series of still images synchronized with a soundtrackrequires manually setting the image frame rate to the tempo of theaudio. Changing the audio to synchronize to the picture requires theuser to edit the sound or manually alter the playback rate. Audio withan inconsistent tempo requires the user to manually determine wheresignificant audio cues reside and edit the still images to thissequence. Any changes in audio require the process to be repeated forthe modified portion.

The ability to quickly, simply and iteratively produce new media contentis of special interest in contexts such as movie making. In particular,home consumer production of movie content suffers from the lack of easyto use yet powerful composition tools that do not require video or audioanalysis by the user. Currently there are no tools that allow the userto easily synchronize or edit a motion picture or series of still imagesto an unrelated audio track or tracks. In addition, there is nomechanism by which pre-existing recordings can be easily and efficientlycombined to present the desired effect. This is particularly true fornon-professionals.

In summary, there is a need for a time-based media processing systemthat is capable of providing high-quality, adaptive media productionswithout requiring a significant level of skill on the part of the user,and is therefore suited for use by both the professional and averageconsumer. There would be great utility in a tool in which two unrelatedmedia signals can be combined to create a motion picture synchronized toa sound track or other media signals and vice versa.

SUMMARY OF THE INVENTION

According to an aspect of the invention a method and a computer readablemedium are provided for resequencing a first input signal, which has afirst playback sequence, based upon at least one characteristic of asecond input signal. The method and medium both include analysis of thecharacteristic of the second input signal, and modifying of the firstplayback sequence of the first input signal to generate a secondplayback sequence. The modification occurs based upon the analysis. Inaddition, a third signal including the second playback sequence isoutput.

In one embodiment, the first input signal is a video signal, the secondinput signal is an audio signal, and the third signal is a video signal.In another embodiment, the first input signal and the second inputsignal are the same signal. It is possible that the analyzing and theoutputting occur in real time. Further, the second input signal can beoutput along with the third signal. The analyzing can occur at afrequency based upon a frame rate of the first input signal. Themodifying can also include altering a frame rate.

In another embodiment, the first input signal is actually multiple inputvideo signals, each having a linear playback sequence, and the thirdsignal is actually multiple output video signals. The characteristic isactually multiple characteristics, each being associated with one of theinput video signals. In this embodiment, the modifying requiresmodifying of a linear playback sequence of each of the input videosignals based upon the analysis of the associated characteristics togenerate an output sequence. The outputting also requires outputtingeach output video signal in accordance with its output sequence.

In another embodiment, the first input signal includes multiple layers,as does the third signal. Similarly, the characteristic includesmultiple characteristics, each being associated with one of the layers.In this embodiment, the modifying includes modifying a layer playbacksequence of each of the layers based upon the analysis of the associatedcharacteristics to generate an output layer sequence. Finally, theoutputting requires outputting of each layer in accordance with itsoutput layer sequence.

According to another aspect, a method and a computer readable medium areprovided for resequencing a first input signal based upon at least onecharacteristic of a second input signal. The first input signal has aseries of frames, each having a frame number. The method and computerreadable medium include analyzing the characteristic of the second inputsignal. They also include calculating, for each frame in the series offrames, an offset based upon the analyzing. The offset is then added tothe frame number to obtain an output frame number. Finally, a thirdsignal is output in a playback sequence using the output frame numbers.

According to another aspect, a method and a computer readable medium areprovided for outputting a signal based upon at least one characteristicof an input signal. The method and medium require analysis of the atleast one characteristic of the input signal. They also require controlof an external device based upon the analysis of the at least onecharacteristic, and output of the signal using the controlled externaldevice.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is further described in the detailed descriptionthat follows, by reference to the noted drawings by way of non-limitingexamples of embodiments of the present invention, in which likereference numerals represent similar parts throughout several views ofthe drawing, and in which:

FIGS. 1A, 1B, and 1C are graphs plotting source video, source audio andoutput video, respectively, against time;

FIG. 2 shows exemplary video frame substitution based upon an audiosignal, according to an aspect of the present invention; and

FIG. 3 is a flow chart showing exemplary processing to determine anoutput stream, according to an aspect of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

In view of the foregoing, the present invention, through one or more ofits various aspects, embodiments and/or specific features orsub-components, is thus intended to bring out one or more of theadvantages as specifically noted below.

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be evident, however, toone skilled in the art that the present invention may be practicedwithout the specific details. In other instances, well-known structuresand devices are shown in block diagram form to facilitate explanation.The description of preferred embodiments is not intended to limit thescope of the appended claims.

An object of the invention is to enable new efficiencies, methods, andforms in the production of media content. The invention also aims tosatisfy a need for a media processing system that reduces the labor andexpertise necessary to create media content. Real time applications ofthis invention can immediately provide visual feedback from audio inputas an essential aid to the production and creation of multimedia.

The invention may be used to help visualize audio segments with nonassociated video segments. Individual tracks can be visualized to helpillustrate the behavior of a musical instrument, a voice or a soundeffect. Multiple images can be shown simultaneously to separate, similarto a graphic equalizer, the complexity and dynamics of an audio segment.In another embodiment, video images may be layered with each layercontaining a separate object. Each layer may be individually controlledby the system so that different objects are graphically manipulated bydifferent discrete media signals.

Aspects of the present invention may be implemented on one or morecomputers executing software instructions, or may be encoded onto readonly memory, PROM, EPROM, firmware, or circuitry for use in anelectronic apparatus. According to one embodiment of the presentinvention, server and client computer systems transmit and receive dataover a computer network or standard telephone line. The steps ofaccessing, downloading, and manipulating the data, as well as otheraspects of the present invention are implemented by central processingunits (CPUs) in the server and client computers executing sequences ofinstructions stored in a memory. The memory may be a random accessmemory (RAM), read only memory (ROM), a persistent storage, such as amass storage device, or any combination of these devices. Execution ofthe sequences of instructions causes the CPU to perform steps accordingto embodiments of the present invention.

The instructions may be loaded into the memory of the server or clientcomputers from a storage device or from one or more other computersystems over a network connection. For example, a client computer maytransmit a sequence of instructions to the server computer in responseto a message transmitted to the client over a network by the server. Asthe server receives the instructions over the network connection, itstores the instructions in memory. The server may store the instructionsfor later execution, or it may execute the instructions as they arriveover the network connection. In some cases, the downloaded instructionsmay be directly supported by the CPU. In other cases, the instructionsmay not be directly executable by the CPU, and may instead be executedby an interpreter that interprets the instructions. In otherembodiments, hardwired circuitry may be used in place of, or incombination with, software instructions to implement the presentinvention. Thus, the present invention is not limited to any specificcombination of hardware circuitry and software, nor to any particularsource for the instructions executed by the server or client computers.

In one embodiment, the invention alters a single media signal or acombination of two or more media components into a modifiedpresentation. In an embodiment, the system analyzes a second media fileand samples a characteristic of the second media file. For example, thesecond media file may be an audio file that is analyzed for any soundsthat exceed a pre-set decibel threshold. In this example, a derivativefile would store all times when the decibel threshold has been exceeded,as well as the amount by which the decibel levels were exceeded. If thesecond media file is a stream, i.e., real time, instead of a file, thederivative file would simply be a single value resulting from a realtime analysis occurring at the sampled frequency.

An output media file is then formed based upon the first media signaland the analysis of the second media signal (i.e., the derivative file).That is, the derivative file is used to modify the output file.Additionally, the derivative file can be used to control an externaldevice containing the first media signal. Examples of this device are adirect disk recorder (DDR), such as TIVO, and a DVD player. Finally, thesecond media file and the output media file are played together.Specifically, when the derivative file so indicates, the systemrearranges the playback sequence of frames of the first media file. Inone embodiment, the output signal is a modified media presentationhaving multiple media components.

An example is now provided with reference to FIGS. 1A, 1B, and 1C inwhich a sequence of frames in a video signal is altered based upon adetected amplitude of an audio signal. FIG. 1A shows standard sourcevideo playback as a linear sequence of frames over time. FIG. 1B showsan amplitude of an audio signal as it varies with time. As shown in FIG.1C, the output signal is created from a non-linear reassembly of framesof the source video so that each frame correlates the amplitude of theaudio with the future/past frames of the source video.

In an audio and video media signal embodiment, the video signal mayinclude moving objects appearing to move in a non-linear manner in syncwith a media, which may be the audio file. For example the movement ofthe objects in the video may appear to move to the beat of the audiotrack.

Normally, audio and video files are played in a specific sequence thatreplicates a recorded source subject, which may be a musical instrumentperformance or a movement of objects. The sequence of sounds replicatesmusic and the sequence of pictures shows a visual representation ofmoving figures. Electronic audio files have a preset sampling rate atwhich specific sounds are sequentially emitted by an audio file player.Similarly, video files also have a preset frame rate at which a specificsequence of video frames are displayed by a media player. Because theaudio sample rate and video frame rate may be fairly high, e.g., 160kbs/second or higher and 30 frames/second, respectively, the human earsand eyes cannot distinguish the breaks between the tones and pictureframes.

As an example, the media signal may be a drum sound that defines thebeat of the song of an audio file. The inventive system can define thedrum sound as the characteristic and sample this characteristic whileplaying the audio file normally. When the audio and video tracks areplayed together, the inventive system will automatically alter thesequence of the video file based upon the characteristic of the audiosignal. Specifically, the video frame that would normally be displayedat the time of the characteristic (i.e., the beat) is not displayed andan alternate substitute video frame is displayed in its place. Inaddition, each and every frame could be reordered.

In other embodiments, any other type of media signal characteristic maybe used to trigger the non-linear playback of a media signal. Forexample, because a video signal may be completely independent of anaudio signal, the picture and sound files used by the system can beassociated or completely disassociated. An example of associated audioand video signals is a musical performance featuring audio and videorecordings of the musical performance. An example of disassociated audioand video files is a song and unrelated video footage of a sportingevent. The system does not rely upon synchronizing markers in both theaudio and video and is therefore compatible with any media signal, suchas an audio file and a video file, as long as one of the files has acharacteristic off which the system can trigger.

In an embodiment, a substitute frame is a frame that is close insequence to the frame that would normally be displayed. For example, thesystem may display a video frame that is a few frames ahead of or behindthe frame that would normally be displayed. This substitution of an outof sequence frame creates an effect where objects appear to be movingnormally in a linear manner and then when the audio signalcharacteristic is detected, a non-sequential video frame or frames aredisplayed between the sequential frames. The non-sequential frames maybe ahead of or behind the normally next sequential frame. If thenon-sequential frame is ahead of the normal sequence, the video willappear to jump forward at the moment of the audio signal characteristic.Similarly, if the non-sequential frame is behind the normal sequenceframe, the video will appear to jump backwards.

In yet another embodiment, the frame or frames displayed when the audiosignal characteristic is detected may be displayed multiple timescreating a temporary freezing of motion. When the audio characteristichas passed, the video returns to its normal linear playback until thenext discrete audio signal is detected. The effect is a pulsation in thenormally linear motion of the video that is synchronized with the audiosignal characteristic.

An embodiment will now be described with reference to FIG. 2. A firstsignal 1, which is a video signal in this example, includes multipleframes. In FIG. 2, frames 11-18 are shown with corresponding framenumbers 11-18 (Frame 10 is only partially shown). A second signal 2 isan audio signal in this example. A characteristic of the audio signal 2,e.g., an amplitude, may be sampled. In FIG. 2, the first three samplingperiods 20, 22, 24 the audio signal has an amplitude of 0. Thus, becauseno characteristic is detected, an output signal 3 is the same as theinput signal 1. In other words, frame numbers 10, 11, and 12 aredisplayed at frames 40, 42, 44. At the time of the fourth sampling 26,an amplitude of 0.4 is detected. Thus, a substitute frame 46 (framenumber 17) is displayed instead of the normally displayed frame 13 ofthe video signal 1. In other words, the output signal 3 displays framenumbers 10, 11, 12 and 17, in this order. Display of frame 17 is aheadof the normal sequence and thus, it will appear that the character hasjumped ahead of where he would be expected to be.

At the fifth and sixth sampling periods 28, 30 the amplitude hasreturned to 0. Thus, the next output frames 48, 50 will again correspondto the video signal 1 at those times (i.e., frame numbers 14 and 15 aredisplayed). At the time of the seventh sampling period 32, an amplitudeof 0.2 is detected. Thus, a substitute frame 52 is displayed instead ofthe normally displayed frame number 16 of the video signal 1. In otherwords, the output signal 3 displays frame numbers 10, 11, 12, 17, 14,15, and 18 in this order. Display of frame 18 is not in the normalsequence and thus, it will appear that the character has again jumpedahead of where he would be expected to be. At the eighth and ninthsampling periods 34, 36 the amplitude has returned to 0. Thus, the nextoutput frames 54, 56 will again correspond to the video signal 1 atthose times.

Various effect modifications can be made to the described system. Forexample, the magnitude of the jump in the non-sequential video frame canbe controlled by the user of the system. This offset can be dynamic andcan be controlled to grow, shrink, oscillate, decay, delay, drift andsustain.

In an alternative embodiment, a single media signal may be used. In thisembodiment, the system detects characteristics in the single mediasignal. The system may alter the output signal of the single mediasignal based upon the detected characteristics in the same media. Forexample in a video signal, any of the video effects may be applied tothe video media upon the detection of a characteristic in the same videosignal. Thus, the system may utilize a single media source to produce analtered output media signal.

In another embodiment, a video frame from a video source other than thesecond video signal may be the substituted frame(s) displayed when theaudio characteristic is detected. In this embodiment, the second videofile may be any other type of visual data including but not limited to:a 2-D file, 3-D data, and a live video feed. When the other video sourceis an animation file, the animation file may be controlled by the systemso that the graphical display is altered when an audio characteristic isdetected.

Because 2-D and 3-D computer graphics programs continuously display acomposite image rather than a series of sequential frames, the inventivesystem will operate in a different manner when applied to such aprogram. For example, the motion of each displayed object may beindividually controlled. Thus, the control of individual components maycorrespond to different characteristics. In an embodiment, a firstobject may be altered by a first characteristic and a second object maybe altered by a second characteristic. For example, the first object mayappear to bounce in response to the first characteristic and the secondobject may be enlarged in response to the second characteristic. Anynumber of visual effects may be applied to the 2-D or 3-D objectsincluding movement, appearance, camera angle, shape, etc.

In another embodiment, the visual display may be a layered composite ofvisual frames. Each of the frame layers may contain a separate objectand each layer may be individually manipulated in response to detectingof a characteristic. By varying only some of the frames of thecomposite, only objects illustrated in the layers are visually altered.For example, the first layer may include a background of mountains, asecond layer may include trees and a lake, a third layer may include anairplane and the fourth layer may include a car. The mountain layer maybe altered by a bass audio signal, the trees and lake layer may bealtered by a mid frequency audio signal. The airplane layer maybealtered by a high frequency audio signal and the car layer may bealtered by a discrete audio signature. The visual effect would showindividual objects appearing to be independently altered by differentcharacteristics.

In yet another embodiment, the frame rate can be altered, so that thedisplay rate is higher or lower than the normal 30 frames/second. Anoffset calculation determined from signal 2 (described below) can beused to vary the playback rate. When a characteristic is detected, thesystem may temporarily accelerate the playback speed to 150frames/second. The accelerated frames may be the normal sequential videoframes, substitute frames or a visual file from another source ormultiple other sources. The visual effect of displaying multiple framesat a faster rate would be acceleration, because the velocity of objectsdisplayed increases. Conversely, if the frame rate is slowed, the visualeffect would be a deceleration of moving objects. Of course, this framerate effect can be applied to a single region visual display, a layeredvisual display or a multiple region visual display.

In another embodiment, multiple frames can be displayed in the timeallotted for a single frame. Similarly, partial frames or blended framesmay be inserted as substitute frames in a sequence of video frames.

One preferred embodiment of the invention is implemented as a set ofsoftware instructions on a computer. In this manifestation, theinvention analyzes an audio signal. that may be either analog ordigital. Exemplary audio formats include stereo, mono, and multipletrack signals, such as MP3, Quicktime, soundedit, MIDI, IMA, WAV, AIFF,Sound designer II, sun .au, or any format that may be developed in thefuture. Sound can be live, pre-recorded, or transmitted through airwavesor dedicated lines, such as the internet, a LAN or a WAN. The audiosignal can contain multiple tracks, and the invention can discernindividual (split) tracks (e.g.,—a drum track residing in a multi-trackcomposition or score). The invention can accept signals frommicrophones, audio tapes (e.g., DAT and cassette), video tapes withsound tracks, CD, DVD, MIDI devices, mini disc or any future soundtechnologies.

The characteristic being detected can also be located in a signalcreated by a user. The user created signal could be employed when theactual audio signal does not provide the intended effect. That is, theuser created signal can be used to achieve the intended effect. In thiscase, the user can select or alter specific characteristics of an audiotrack, for example its tempo, tone, beat, pitch, duration, carrierfrequency, scale or volume. The user can also adjust the amplitudes ofspecific sound frequencies and can shift the pitch of the audio. Inaddition, the user can generate a tone (by specifying a frequency,amplitude and shape) and can insert the tone at any point in time. Sucha user created signal can be recorded or can be live. An alternativeuser-created signal can include a duplicate of an existing audio track,the duplicate being subject to a user selected audio filtration. In thiscase, the original unmodified audio track and a modified track can bothexist. After the user created signal has been created, it can be mixedwith another audio track, e.g., the un-modified track. Alternatively,the user created audio signal is not played back or encoded into thefinal audio signal; rather, its data is merely used to generate timebased video effects.

When the first component is a picture, it can include still images, afilm or video. The image can be received by the invention in analog ordigital format and can be live, broadcast, or transmitted. The picturesignal can be, for example, YUV, YC, RGB, component, composite or IEEE1394. The signal could be digital video stored on a local or remotecomputer. The video can also reside on a pre-recorded medium, such as aDVD, VCD, TIVO, DDR, MiniDV, laserdisc, digibeta, d1, d3 or any mediumor apparatus that records images.

In an alternative embodiment, the system may analyze a video file tocreate an output audio file, in which the video file plays normally(sequentially) but the audio file is modified when certaincharacteristics are detected. The system may then play the video in anormal sequential manner and when the video characteristics aredetected, the audio signal is altered. The audio signal modification caninclude: volume, tempo, frequency, sequence, distortion, tone or anyother audio effect. The effect is to view a normal video file while theaudio file is periodically distorted based upon the discrete videosignal.

One implementation of the invention re-sequences motion picture framesbased upon a secondary media signal. This re-sequencing can be conductedin real time, utilizing live or pre-recorded audio or live orpre-recorded video. Playback can occur in real time, and/or can bestored for later playback.

The inventive system is particularly useful in conjunction with livemedia. In such a case, the media signals processed by the system mayinclude live and/or recorded audio and live and/or recorded video. Byinputting the live media data, the system can process the media data toincorporate the substitute component or a media effect by utilizingpreviously recorded substitute data when a characteristic is detected.Visual effects that can be implemented during a detection of thecharacteristic can include: varying the camera angle, color, rotation,zoom, pixelation, etc. Alternatively, if multiple cameras are recordingthe same live performance, the signals received from the differentcameras can act as the substitute media signals and data from a second,third, fourth, etc. media source can be inserted when the characteristicis detected. The inventive processing of live media can be configured tooutput live audio with recorded video, live video with recorded audio,or live video with live audio. Of course any other combination wouldalso be within the purview of the present invention.

Two generalized equations that describe the re-sequencing embodimentwhen utilizing two media signals are shown below.

(1) O(i) = O_(max) F(S₂(i)) Offset calculation (2) S₃(i) = S₁(O(i) + i)Output media Where: S₁(i) = Media signal 1 0 ≦ i ≦ (n₁ − 1) S₂(i) =Media signal 2 0 ≦ i ≦ (n₂ − 1) S₃(i) = Output signal 0 ≦ i ≦ (n₁ − 1)F(i) = Filter Function Real Number O(i) = offset at time i 0 < |O(i)| <O_(max) O_(max) = maximum offset (set by user)

S₁ and S₂ represent independent media signals that are used by thealgorithm to produce an output signal S₃. As discussed, the mediasignals can be virtually any type of media. In the followingdescription, S₁ represents a video signal and S₂ represents an audiosignal. The “i” represents the sequential indicator. The filter functionF can be any linear or non-linear equation that produces a real number.O_(max) is the maximum offset and can be any number that is set by theuser.

Equation (1) is the offset calculation. The offset calculation solvesthe offset based upon the filter function of S₂(i) and has a value whena user designated characteristic is detected in S₂(i). If acharacteristic is not detected the filter function does not produce anoutput and F(S₂(i))=0. Consequently, O(i)=O_(max) F(S₂(i))=0. If acharacteristic is detected by the filter function, the filter functiongenerates a real number that is multiplied by O_(max) to produce a valuefor O_(i). In a preferred embodiment, the O(i) value is an integer.

Equation (2) solves the output media S₃(i). If a characteristic is notdetected O(i)=0 and the output media signal is the same as thecorresponding media signal 1, S₃(i)=S₁(0+i)=S₁(i). If a characteristicis detected, O(i) has a value, which may be positive or negative. TheO(i) value is added to i to produce an offset value that indicates aframe ahead of or behind the sequential marker “i”. Thus, duringplayback the output media will be the same as media signal 1 until adesignated characteristic is detected in media signal 2. When thecharacteristic is detected, the filter function will produce a value andthe output media will be offset from media signal 1. The output mediamay be output alone or synchronously with media signal 2 or any othermedia signal.

An example of a common manifestation of the invention is now described.In this example, the invention is implemented as a set of softwareinstructions on a computer and two media signals, one audio (S₂) and onevideo (S₁), are stored on the hard drive of a computer. The two mediasignals (i.e., the video and the sound) are combined and displayed as anew media composition having a sequence of frames, each containingdigital data describing, for example, the audio or visual content of theframe. When the composition is played, the frames are output at acomposition frame rate. The composition includes the source audio track(S₂) and an output video track (S₃).

Referring to FIG. 3., the user, with an input device such as a mouse orkeyboard, selects the video source (S₁) and the audio source (S₂), atsteps S2 and S4, respectively, and places them on a time line. Ofcourse, more than two sources can be selected and processed but for thesake of simplicity the example only uses two signals.

The user also selects a characteristic of the audio signal, at step S6.If no characteristic is selected, default settings for audio processingare used. An example of an audio characteristic is a frequency rangelevel, such as 50-100 hz. The audio characteristic can be sampled in adirect temporal relationship to the frame rate.

At step S8, the user enters a value for the maximum offset. Thisvariable determines the maximum range from the current frame theinvention will use to select a substitute frame. In other words, thisvalue will determine the maximum offset between the source frame and thecomposition frame to be displayed. Although this variable is describedas a fixed value, this number can also be variable. Specifically, themaximum offset can be a function of any input including a characteristicof the media signal. In other embodiments, the maximum offset can beturned off, disabling the inventive media effects system in the eventthat normal playback is desired.

At step S10, the filtration function occurs. That is, the audio source(S₂) is analyzed for the designated characteristic. The filtering mayoccur at a user selected sampling frequency. The sampling frequencydefaults to the frame rate of the source picture component (S₁): forNTSC television this is approximately 30 times a second.

Based upon the filtering function, a data set is determined. Morespecifically, at intervals defined by the sampling frequency, the audiocharacteristic is sampled and normalized. The normalization occurs sothat the substitute frame will not be more than the user selectedmaximum offset.

In the invention's real time embodiment, step S10 is executed inparallel with steps S16-S18. At step S16, the offset O(i) is calculatedbased upon equation (1). Subsequently, the output signal is calculatedbased upon equation (2). After step S18, it is determined whetherprocessing should continue. If so, the process returns to step S10 andrepeats. If the processing is determined to be completed, the processingends at step S22.

In an example, a new media composition at 30 frames per second iscreated. The new media composition includes two media signals. The firstmedia source signal (S₁) is a series of 300 images on a hard drive on acomputer. The native playback rate of the motion picture sequence is 30frames per second. The set S₁(i) contains each frame of the series,S₁(1) being the first frame and S₁(300) being the last frame.

The second media source signal (S₂) is an audio track 10 seconds long.Played from the beginning, the track contains 3 seconds of silence, a 60hz tone with 100% amplitude (level), lasting 1/30^(th) of a second, 3seconds of silence, a 60 hz tone with 50% amplitude lasting 1/30^(th) ofa second, and 3 and 28/30^(th) seconds of silence. S₂(i) contains a setof 300 values.

In this example after the signals are selected at steps S2 and S4, atstep S6 the user selects bass tones having a frequency range of 50-70 hzas the audio characteristic. The default 30 frames per second is usedfor the composition frame rate. At step S8, the user selects the maximumoffset to be 60 frames.

At step S10 the filtering function processes the 300 members of S₂(i) bysampling the audio characteristic. In this example, the audiocharacteristic is not detected when i=1 through 90. Thus, F(S₂(1))through F(S₂(90))=0. At i=91, F(S₂(91))=100, thus normalizedF(S₂(91))=1. At i=92-182, the audio characteristic is not detected andF(S₂(92)) through F(S₂(182)) are 0. At i=183, F(S₂(i))=60, thusnormalized F(S₂(183))=0.6. At i=184 through 300, F(S₂(i))=0 and thusF(S₂(184)) through F(S₂(300)) are 0.

Although the values are described as being normalized between 0 and 1,it is also possible to use a threshold or non-linear function todetermine the F(S₂(i)) values. In a threshold embodiment, the equationsmay be F(S₂(i))=0 if the detected characteristic has a value<50 andF(S₂(i))=1 if the detected characteristic has a value≧50. Thus, if thedetected characteristic =60 then F(S₂(i))=1. Similarly, any other linearor non-linear formula may be used to define F(S₂(i)) as a function ofthe detected characteristic.

The output media and offset equations (i.e., equations (1) and (2)) arethen solved. As described above, it is preferable to use integers, thus,the INTEGER function is employed. At i=1 through 90, S₃(i)=S₁ (INTEGER(0*60+i))=S₁(i). At i=91, S₃(91)=S₁(INTEGER (1*60+91))=S₁(151). At i=92through 182, S₃(i)=S₁(i). At i=183, S₃(183)=S₁(INTEGER(0.6*60+183))=S₁(219).

The new composition sequence, C(i), is created. It includes 300 imagesplayed sequentially. C(i) contain the same frames as S₁(i), with theexception that C(91) would play corresponding frame S₁(151) and C(183)would contain S₁(219).

Table 1 indicates the values of the variables described in the exampleabove.

TABLE 1 Characteristic i value F(S(i)) O_(max) O(i) C(i)  1-90 0 0 60 1-90 PS(i)  91 100  1 60 151 PS(151)  92-182 0 0 60  92-182 PS(i) 18360  0.6 60 219 PS(219) 184-300 0 0 60 184-300 PS(i)

Thus, the sequence of frames in the new media composition is:S(1)-S₁(90), S₁(151), S1(92)-S₁(182), S₁(219), S₁(184)-S₁(300). Notethat S₁(151) and S₁(219) are displayed twice and frames S₁(91) andS₁(183) are not displayed at all.

Although the invention has been described with reference to severalexemplary embodiments, it is understood that the words that have beenused are words of description and illustration, rather than words oflimitation. Changes may be made within the purview of the appendedclaims, as presently stated and as amended, without departing from thescope and spirit of the invention in its aspects. Although the inventionhas been described with reference to particular means, materials andembodiments, the invention is not intended to be limited to theparticulars disclosed; rather, the invention extends to all functionallyequivalent structures, methods, and uses such as are within the scope ofthe appended claims.

What is claimed is:
 1. A method for modifying a first input signalcomprising a video signal based upon at least one characteristic of asecond input signal comprising an audio signal, the method comprising:analyzing the at least one characteristic of the second input signal;altering a frame rate of a portion of the first input signal based uponthe analysis of the at least one characteristic; and outputting a thirdvideo signal comprising the first input video signal including theportion with the altered frame rate.
 2. The method of claim 1, in whichthe analyzing and outputting occur in real time.
 3. The method of claim1, in which the outputting further comprises outputting the second inputsignal along with the third signal.
 4. The method of claim 1, in whichthe analyzing occurs at a frequency based upon a frame rate of the firstinput signal.
 5. The method of claim 1, in which the first input signalcomprises a plurality of input video signals; in which the third signalcomprises a plurality of output video signals; in which the at least onecharacteristic comprises a plurality of characteristics, eachcharacteristic being associated with one of the input video signals; inwhich the modifying further comprises modifying each of the input videosignals based upon the analysis of the associated characteristics; andin which the outputting further comprises outputting each modifiedoutput video signal.
 6. The method of claim 1, in which the first inputsignal comprises a plurality of layers; in which the third signalcomprises the plurality of layers; in which the at least onecharacteristic comprises a plurality of characteristics, eachcharacteristic associated with one of the layers; in which the modifyingfurther comprises modifying each of the layers based upon the analysisof the associated characteristics; and in which the outputting furthercomprises outputting each modified layer.
 7. A method for resequencing afirst input signal, which has a series of frames, each having a framenumber, based upon at least one characteristic of a second input signal,the method comprising: analyzing the at least one characteristic of thesecond input signal; for each frame in the series of frames, calculatingan offset based upon the analyzing, the offset being added to the framenumber to obtain an output frame number; and outputting a third signalin a playback sequence using the output frame numbers.
 8. The method ofclaim 7, in which the first input signal comprises a video signal, thesecond input signal comprises an audio signal, and the third signalcomprises a video signal.
 9. The method of claim 7, in which theanalyzing and outputting occur in real time.
 10. The method of claim 7,in which the calculating further comprises calculating based upon a userdefined maximum offset.
 11. The method of claim 10, in which the offsetdynamically varies.
 12. The method of claim 8, in which the first inputsignal comprises a plurality of input video signals each having a linearplayback sequence; in which the third signal comprises a plurality ofoutput video signals; in which the at least one characteristic comprisesa plurality of characteristics, each characteristic being associatedwith one of the input video signals; in which the calculating furthercomprises calculating an offset for each of the output video signalsbased upon the analysis of the associated characteristics to generate anoutput frame number for each output video signal; and in which theoutputting further comprises outputting each output video signal inaccordance with its output frame numbers.
 13. The method of claim 8, inwhich the first input signal comprises a plurality of input layers; inwhich the third signal comprises the plurality of output layers; inwhich the characteristic comprises a plurality of characteristics, eachcharacteristic associated with one of the output layers; in which thecalculating further comprises calculating an offset for each of theoutput layers based upon the analysis of the associated characteristicsto generate an output frame number for each output layer; and in whichthe outputting further comprises outputting each output layer inaccordance with its output layer sequence.
 14. A method for outputtingan output video signal, comprising a series of frames, based upon atleast one characteristic of an input audio signal, the methodcomprising: analyzing the at least one characteristic of the input audiosignal; for each frame in the series of frames, calculating an offsetbased upon the analyzing, the offset being added to the frame number toobtain an output frame number; controlling an external device based uponthe analysis of the at least one characteristic; and outputting theoutput video signal using the controlled external device in a playbacksequence using the output frame numbers.
 15. The method of claim 14, inwhich the analyzing and outputting occur in real time.
 16. A computerreadable medium storing a computer program for modifying a first inputsignal comprising a video signal, based upon at least one characteristicof a second input signal comprising an audio signal, the mediumcomprising: an analyzing source code segment that analyzes the at leastone characteristic of the second input signal; a modifying source codesegment that modifies a frame rate of a portion of the first inputsignal based upon the analysis of the at least one characteristic; andan output source code segment that outputs a third video signalcomprising the first input video signal including the portion with thealtered frame rate.
 17. The medium of claim 16, in which the analyzingsource code segment analyzes at a frequency based upon a frame rate ofthe first input signal.
 18. The medium of claim 16, in which theanalyzing source code segment and the output source code segment processin real time.
 19. The medium of claim 16, in which the output sourcecode segment further comprises outputting the second input signal alongwith the third signal.
 20. The medium of claim 16, in which the firstinput signal comprises a plurality of input video signals; in which thethird signal comprises a plurality of output video signals; in which theat least one characteristic comprises a plurality of characteristics,each characteristic being associated with one of the input videosignals; in which the modifying source code segment further comprisesmodifying each of the input video signals based upon the analysis of theassociated characteristics; and in which the output source code segmentfurther comprises outputting each modified output video signal.
 21. Themedium of claim 16, in which the first input signal comprises aplurality of layers; in which the third signal comprises the pluralityof layers; in which the at least one characteristic comprises aplurality of characteristics, each characteristic associated with one ofthe layers; in which the modifying source code segment further comprisesmodifying each of the layers based upon the analysis of the associatedcharacteristics; and in which the output source code segment furthercomprises outpuffing each modified layer.
 22. A computer readable mediumstoring a computer program for resequencing a first input signal, whichhas a series of frames, each having a frame number, based upon at leastone characteristic of a second input signal, the medium comprising: ananalyzing source code segment that analyzes the at least onecharacteristic of the second input signal; a calculating source codesegment that, for each frame in the series of frames, calculates anoffset based upon the analysis, the offset being added to the framenumber to obtain an output frame number; and an output source codesegment that outputs a third signal in a playback sequence using theoutput frame numbers.
 23. The medium of claim 22, in which the firstinput signal comprises a video signal, the second input signal comprisesan audio signal, and the third signal comprises a video signal.
 24. Themedium of claim 22, in which the analyzing source code segment and theoutput source code segment process in real time.
 25. The medium of claim22, in which the calculating source code segment further comprisecalculating based upon a user defined maximum offset.
 26. The medium ofclaim 25, in which the offset dynamically varies.
 27. The medium ofclaim 23, in which the first input signal comprises a plurality of inputvideo signals each having a linear playback sequence; in which the thirdsignal comprises a plurality of output video signals; in which the atleast one characteristic comprises a plurality of characteristics, eachcharacteristic being associated with one of the input video signals; inwhich the calculating source code segment further comprises calculatingan offset for each of the output video signals based upon the analysisof the associated characteristics to generate an output frame number foreach output video signal; and in which the output source code segmentfurther comprises outputting each output video signal in accordance withits output frame numbers.
 28. The medium of claim 23, in which the firstinput signal comprises a plurality of input layers; in which the thirdsignal comprises the plurality of output layers; in which thecharacteristic comprises a plurality of characteristics, eachcharacteristic associated with one of the output layers; in which thecalculating source code segment further comprises calculating an offsetfor each of the output layers based upon the analysis of the associatedcharacteristics to generate an output frame number for each outputlayer; and in which the output source code segment further comprisesoutputting each output layer in accordance with its output layersequence.
 29. A computer readable medium storing a computer program foroutputting an output video signal, comprising a series of frames, basedupon at least one characteristic of an input audio signal, the mediumcomprising: an analyzing source code segment that analyzes the at leastone characteristic of the input signal; an offset source code segment,for calculating for each frame in the series of frames, an offset basedupon the analyzing, the offset being added to the frame number to obtainan output frame number; a controlling source code segment that controlsan external device based upon the analysis of the at least onecharacteristic; and an output source code segment that outputs theoutput signal using the controlled external device in a playbacksequence using the output frame numbers.
 30. The medium of claim 29, inwhich the analyzing source code segment and the output source codesegment occur in real time.
 31. A method for resequencing a first inputsignal, comprising a series of frames, back based upon at least onecharacteristic of a second input signal, the method comprising:analyzing the at least one characteristic of the second input signal;for each frame in the series of frames, calculating an offset based uponthe analyzing, the frame number being altered based upon the offset toobtain an output frame number; and outpuffing a third signal in aplayback sequence using the output frame numbers.
 32. A computerreadable medium storing a computer program for resequencing a firstinput signal, comprising a series of frames, based upon at least onecharacteristic of a second input signal, the medium comprising: ananalyzing source code segment that analyzes the at least onecharacteristic of the second input signal; an offset calculating sourcecode segment that calculates, for each frame in the series of frames, anoffset based upon the analyzing, the frame number being altered basedupon the offset to obtain an output frame number; and an output sourcecode segment that outputs a third signal in a playback sequence usingthe output frame numbers.
 33. The method of claim 1, in which the firstinput signal comprises a plurality of input signals.
 34. The medium ofclaim 16, in which the first input signal comprises a plurality of inputsignals.
 35. An apparatus for resequencing a first input signal, whichhas a series of frames, each having a frame number, based upon at leastone characteristic of a second input signal, the apparatus comprising:an analyzer that analyzes the at least one characteristic of the secondinput signal; a calculator that calculates, for each frame in the seriesof frames, an offset based upon the analyzing, the frame number beingaltered based upon the offset to obtain an output frame number; and anoutput device that outputs a third signal in a playback sequence usingthe output frame numbers.
 36. An apparatus for resequencing a firstinput signal, which has a series of frames, each having a frame number,based upon at least one characteristic of a second input signal, theapparatus comprising: an analyzer that analyzes the at least onecharacteristic of the second input signal; a calculator that calculates,for each frame in the series of frames, an offset based upon theanalyzing, the offset being added to the frame number to obtain anoutput frame number; and an output device that outputs a third signal ina playback sequence using the output frame numbers.
 37. An apparatus formodifying a first input video signal based upon at least onecharacteristic of an input audio signal, the apparatus comprising: ananalyzer that analyzes the at least one characteristic of the inputaudio signal; a frame rate modification system that alters a frame rateof a portion of the first input video signal based upon the analysis ofthe at least one characteristic; and an output device that outputs asecond video signal comprising the first input video signal includingthe portion with the altered frame rate.